A General-Purpose Compression Scheme for Databases

Current adaptive compression schemes such as gzip and compress are impractical for database compression as they do not allow random-access to individual records. The sequitur scheme of Nevill-Manning and Witten also adaptively compresses data, achieving excellent compression but with signiicant main-memory requirements. A preliminary version of sequitur used a semi-static modeling approach to achieve slightly worse compression than the adaptive approach. We describe a new variant of the semi-static sequitur algorithm, ray, that reduces main-memory use and is a candidate for general-purpose compression and random-access to databases. We show that ray achieves better compression than an eecient Huumann scheme and popular adaptive compression techniques.